Bayesian Shrinkage Variable Selection

We introduce a new Bayesian approach to the variable selection problem which we term Bayesian Shrinkage Variable Selection (BSVS ). This approach is inspired by the Relevance Vector Machine (RVM ), which uses a Bayesian hierarchical linear setup to do variable selection and model estimation. RVM is typically applied in the context of kernel regression although it is also suitable in the standard regression context. Extending the RVM algorithm, we include a proper prior distribution for the precisions of the regression coefficients, v j ∼ f(v −1 j |η), where η is a scaler hyperparameter. Based upon this model, we derive the full set of conditional distributions for parameters as would typically be done when applying Gibbs sampling. However, instead of simulating samples from the joint posterior distribution in order to estimate the posterior means of the parameters, we use the full conditionals in order to find the joint maximum of the posterior distribution p(β, σ2,V|y, η) given the value of the hyper-parameter η. While the models with η = 0 result in an “RVM -like” solution, those with η > 0 reinforce further shrinkage leading to more parsimonious models with smaller MSE and prediction errors than traditional RVM models. η is estimated via maximizing the marginal likelihood, i.e. Copyright c © 2007 Artin Armagan and Russell L. Zaretzki Ph.D. Student in Statistics, The University of Tennessee, Knoxville, USA Assistant Professor in Statistics, The University of Tennessee, Knoxville, USA